Papers with LLM-based evaluators
Divide-Conquer-Reasoning for Consistency Evaluation and Automatic Improvement of Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality and consistency of text generated by Large Language Models are not effective. |
| Approach: | They propose a divide-conquer-reasoning approach to evaluate LLM-generated texts using a split-and-conquers evaluator and an automatic metric converter to facilitate this approach. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by a large margin on multiple benchmarks and reduces 90% of output inconsistencies in one iteration. |
Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts (2026.tacl-1)
Copied to clipboard
| Challenge: | Evaluating natural language generation systems is challenging due to the diversity of valid outputs. |
| Approach: | They propose an inversion learning method that learns effective reverse mappings from model outputs back to their input instructions. |
| Outcome: | The proposed method requires only a single evaluation sample and eliminates manual prompt engineering. |
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)
Copied to clipboard
| Challenge: | Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. |
| Approach: | They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. |
| Outcome: | The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines. |
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)
Copied to clipboard
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. |
| Approach: | They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
| Outcome: | The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notes (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing automated metrics fail to align with real-world physician preferences. |
| Approach: | They propose a pipeline that distills real user feedback into structured checklists for note evaluation that are interpretable, grounded in human feedback, and enforceable by LLM-based evaluators. |
| Outcome: | The proposed checklist outperforms baseline evaluations in coverage, diversity, and predictive power for human ratings. |
DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality of machine-generated texts have a relatively low correlation with human performance. |
| Approach: | They propose an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil’s Advocate. |
| Outcome: | The proposed evaluation framework outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. |
Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses (2025.emnlp-industry)
Copied to clipboard
| Challenge: | DeCE is model-agnostic and domain-general, requiring no predefined taxonomies or handcrafted rubrics. |
| Approach: | They propose a decomposed LLM evaluation framework that separates accuracy and recall from accuracy and relevance. |
| Outcome: | The proposed framework achieves stronger correlation with expert judgments than traditional metrics and pointwise LLM scoring. |
Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. |
| Approach: | They use large language models (LLMs) for automatic evaluation to evaluate a sample . they propose several recommendations for integrating LLMs into future classroom evaluations . |
| Outcome: | The proposed model is able to output high scores without meeting the evaluation instructions, the authors note . their model is not able for students to manipulate the model to output specific strings, they say . |
METAL: Towards Multilingual Meta-Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models excel on many standard NLP benchmarks. |
| Approach: | They propose a framework for end-to-end evaluation of Large Language Models as evaluators in multilingual scenarios. |
| Outcome: | The proposed framework evaluates LLMs as evaluators in multilingual scenarios. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
Likelihood-based Mitigation of Evaluation Bias in Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. |
| Approach: | They propose to use LLMs to evaluate sentences with higher likelihoods and lower likelihoods to mitigate the likelihood bias. |
| Outcome: | The proposed method overrates sentences with higher likelihoods while underrating sentences with lower likelihoods. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition (2024.acl-long)
Copied to clipboard
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Large language models (LLMs) are a promising alternative to expensive human evaluations. |
| Approach: | They propose a framework that iteratively aligns LLM-based evaluators with human preference . they decompose a given evaluation task into finer-grained criteria . |
| Outcome: | The proposed framework iteratively aligns LLM-based evaluators with human preference . it decomposes a given evaluation task into finer-grained criteria . the framework is efficient to train and more explainable than relying solely on prompts . |
Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) for large language models typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling. |
| Approach: | They propose a mixed-data approach to training large language models with varying reward clarity . they combine Multiple-choice questions (MCQs) with OE questions for which they use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators. |
| Outcome: | The mixed-data approach improves medical question-answering performance across model scales. |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
Split and Merge: Aligning Position Biases in LLM-based Evaluators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise as automated evaluators for assessing the quality of answers generated by AI systems. |
| Approach: | They propose an alignment-based system that calibrates position bias in a lightweight yet effective manner by taking into account both length and semantics and combining them into a single prompt. |
| Outcome: | Extensive experiments with six LLMs on 11,520 answer pairs show that PORTIA significantly improves consistency and consistency rates with humans. |
CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges (2025.acl-long)
Copied to clipboard
| Challenge: | Empirical evaluations of large language models demonstrate that they improve performance in a wide range of tasks. |
| Approach: | They propose a label-free method for mitigating selection bias during inference by reformulating debiasing as an optimization task. |
| Outcome: | The proposed method mitigates selection bias and improves performance compared to existing methods. |
Re-Evaluating Evaluation for Multilingual Summarization (2024.emnlp-main)
Copied to clipboard
Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, Ellie Pavlick
| Challenge: | Existing studies have shown that automated evaluation approaches correlate with human ratings in English, but this is unclear for other languages. |
| Approach: | They construct a small-scale pilot dataset containing article-summary pairs and human ratings in English, Chinese and Indonesian to measure the strength of summaries. |
| Outcome: | The results show that standard metrics are unreliable measures of quality in Chinese and Indonesian. |
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
Towards AI-Assisted Psychotherapy: Emotion-Guided Generative Interventions (2025.emnlp-main)
Copied to clipboard
Kilichbek Haydarov, Youssef Mohamed, Emilio Goldenhersch, Paul OCallaghan, Li-jia Li, Mohamed Elhoseiny
| Challenge: | Large language models (LLMs) lack rich non-verbal emotional cues essential to real-world therapy. |
| Approach: | They propose a multimodal dataset of 1,441 publicly sourced therapy session videos containing both dialogue and non-verbal signals such as facial expressions and vocal tone. |
| Outcome: | The proposed model improves the quality of generated interventions and evaluators misalign with expert assessments in this domain, highlighting the need for human-centered evaluation. |